Scientists have discovered a possible driving force behind some of nature's most beautiful displays paving the way for more complex and autonomous AI. Researchers wanted to replicate the basic mechanisms behind some of the most highly organised patterns seen in the animal kingdom such as huge swirling starling murmurations and immense twisting herring shoals. Groups like these, consisting in many cases of hundreds of thousands of individual animals, appear to move as if compelled by a collective intelligence, said lead author Dr Marco Mazza, a Lecturer in Applied Mathematics, at Loughborough University. But in reality, could be down to basic survival instincts. "The beauty of nature has inspired artists, philosophers, and scientists for as long as we can remember," said Dr Mazza.

Continuous symmetries and their breaking play a prominent role in contemporary physics. Effective low-energy field theories around symmetry breaking states explain diverse phenomena such as superconductivity, magnetism, and the mass of nucleons. We show that such field theories can also be a useful tool in machine learning, in particular for loss functions with continuous symmetries that are spontaneously broken by random initializations. In this paper, we illuminate our earlier published work (Bamler & Mandt, 2018) on this topic more from the perspective of theoretical physics. We show that the analogies between superconductivity and symmetry breaking in temporal representation learning are rather deep, allowing us to formulate a gauge theory of `charged' embedding vectors in time series models. We show that making the loss function gauge invariant speeds up convergence in such models.

Many loss functions in representation learning are invariant under a continuous symmetry transformation. As an example, consider word embeddings (Mikolov et al., 2013b), where the loss remains unchanged if we simultaneously rotate all word and context embedding vectors. We show that representation learning models with a continuous symmetry and a quadratic Markovian time series prior possess so-called Goldstone modes. These are low cost deviations from the optimum which slow down convergence of gradient descent. We use tools from gauge theory in physics to design an optimization algorithm that solves the slow convergence problem. Our algorithm leads to a fast decay of Goldstone modes, to orders of magnitude faster convergence, and to more interpretable representations, as we show for dynamic extensions of matrix factorization and word embedding models. We present an example application, translating modern words into historic language using a shared representation space.